Method and system for pixel super-resolution of multiplexed holographic color images

ABSTRACT

A method of generating a color image of a sample includes obtaining a plurality of low resolution holographic images of the sample using a color image sensor, the sample illuminated simultaneously by light from three or more distinct colors, wherein the illuminated sample casts sample holograms on the image sensor and wherein the plurality of low resolution holographic images are obtained by relative x, y, and z directional shifts between sample holograms and the image sensor. Pixel super-resolved holograms of the sample are generated at each of the three or more distinct colors. De-multiplexed holograms are generated from the pixel super-resolved holograms. Phase information is retrieved from the de-multiplexed holograms using a phase retrieval algorithm to obtain complex holograms. The complex hologram for the three or more distinct colors is digitally combined and back-propagated to a sample plane to generate the color image.

RELATED APPLICATION

This Application is a continuation of U.S. application Ser. No.16/300,546, filed on Nov. 9, 2018, now issued as U.S. Pat. No.10,795,315, which is a U.S. National Stage filing under 35 U.S.C. § 371of International Application No. PCT/US2017/031984, filed May 10, 2017,which claims priority to U.S. Provisional Patent Application No.62/334,671 filed on May 11, 2016, which are hereby incorporated byreference. Priority is claimed pursuant to 35 U.S.C. §§ 119, 371 and anyother applicable statute.

TECHNICAL FIELD

The technical field generally relates methods and devices for obtainingcolored, microscopic images obtained from holographic images generatedfrom multiple different wavelengths. In particular, the technical fieldrelates to using pixel super-resolution in conjunction with colorde-multiplexing for simultaneous multiplex illumination.

BACKGROUND

Computational microscopy modalities are becoming more and more powerfulthanks to the rapid improvements in digital imaging chips, graphicsprocessing units as well as emerging image reconstruction methods thatenable high-resolution imaging over large sample areas and volumes.Among these different computational microscopy techniques, digitalholography is one of the most widely explored modalities as it permitshigh-throughput 3D imaging of phase and amplitude information ofspecimen. Holographic microscopy in general demands spatial and temporalcoherence of illumination, although partially-coherent or evenincoherent sources can also be utilized in certain imaging designs. Toachieve color imaging in digital holography various methods have beenemployed. One of the most commonly used approaches captures threeholograms at different wavelengths sequentially, at red (e.g., 610-650nm), green (e.g., 520-560 nm) and blue (e.g., 450-480 nm) parts of thespectrum, and then digitally cross-registers and combines theseholograms to reconstruct a color image of the specimen. The sequentialillumination method requires additional time to acquire images at thedifferent wavelengths. In addition, each sequential imaging operationgenerates data corresponding to the particular illumination wavelength;making the sequential process data-intensive.

As an alternative to this sequential color illumination method,simultaneous multi-wavelength illumination of the sample has also beenutilized in combination with a color image sensor chip (e.g., with aBayer color-filter array, CFA) to digitize the resulting multi-colorhologram in one snap-shot. Using the known transmission spectra of thered (R), green (G) and blue (B) filters of the Bayer CFA, three sets ofholograms corresponding to three unique wavelengths can be digitallyretrieved through an inverse mapping (i.e., de-multiplexing) algorithm.Compared to sequential color illumination, this simultaneousillumination approach saves experimental time through digitalde-multiplexing of color channels; however, the reconstructed colorimages are lower resolution and exhibit color artifacts. Unlike naturalimages, holograms contain rapidly changing oscillations/fringes andbecause different channels of the color filters of a Bayer pattern arenot exactly at the same spatial location, the traditional Bayerdemosaicing process, when dealing with the sharp oscillations of ahologram, causes severe fringe artifacts, which become even morenoticeable for wide-field holographic imaging systems with largeeffective pixels or small magnifications. To better handle such samplingartifacts, different Bayer demosaicing approaches have also beenproposed, however, these methods still suffer from the problem ofcreating an artifact-free de-multiplexing of holographic high frequencyfringes created by multi-wavelength illumination.

SUMMARY

In one embodiment, to address the sampling and de-multiplexing relatedchallenges noted above in holographic color imaging, a newhigh-resolution color microscopy technique is introduced that is termedDemosaiced Pixel Super-Resolution (D-PSR). In this D-PSR approach, aplurality of raw holograms a first captured on a Bayer color imagesensor chip (or other color image sensor with CFA) using simultaneous ormultiplexed multi-wavelength illumination, where the sensor plane, thesample, or the light source is shifted by small (sub-pixel) incrementsin the x and y directions (generally parallel to the plane of the activesurface of the color image sensor chip). Pixel super-resolution is thenperformed based on these sub-pixel shifted raw holograms to digitallysynthesize smaller “effective” pixels (e.g., by a factor of ˜3 fold) foreach color element of the Bayer CFA. Using the pre-calibrated spectralcross-talk matrix of each filter of the Bayer CFA at the selectedillumination wavelengths, the three color channels are thende-multiplexed, each of which is also pixel super-resolved. Complexprojection images are digitally reconstructed using an iterative phaserecover process which can be used to back propagate to the object orsample plane to generate the final color image. This D-PSR approachsolves Bayer CFA related spatial sampling limitations and colorartifacts of previous color de-multiplexing approaches, significantlyimproving the performance of holographic high-resolution color imaging.

For experimental demonstration of the D-PSR approach lens-freeholographic on-chip imaging was selected, where the sample is placed onthe top of or adjacent to a Bayer color image sensor chip, typically ata distance of ˜0.3-1 mm away from the chip surface. In this unitmagnification transmission imaging set-up on a chip, the samplefield-of-view (FOV) is equal to the active area of the color imagesensor chip, which is typically ˜20-30 mm² using a state-of-the-art CMOScolor image sensor chip. As a result of this unique imagingconfiguration, the FOV and resolution are decoupled from each other andpartially coherent sources can be utilized to push the resolution of thereconstructed holograms to the diffraction limit. Another importantadvantage of this on-chip holographic imaging approach is thecompactness and cost-effectiveness of its set-up, which makes it highlysuitable for telemedicine applications and field use. Since this is anin-line holographic imaging geometry, the twin-image noise that ischaracteristic of an in-line set-up needs to be eliminated; amulti-height based phase retrieval approach was used for this purpose.D-PSR achieves a color imaging performance that is comparable tosequential illumination of the sample at three distinct wavelengths(corresponding to R, G and B channels) and therefore improves theoverall speed of holographic color imaging. Finally, it should beemphasized that this D-PSR technique is broadly applicable to anyholographic microscopy application (lens-based or lens-free), wherehigh-resolution imaging and simultaneous multi-wavelength illuminationare sought.

In one embodiment, a method of generating a color image of a sampleincludes the operations of obtaining a plurality of low resolutionholographic images of the sample using a color image sensor that has acolor filter array (CFA), the sample illuminated simultaneously byelectromagnetic radiation or light from three or more distinct colors,wherein the illuminated sample casts sample holograms on the color imagesensor and wherein the plurality of low resolution holographic imagesare obtained by relative x, y, and z directional shifts between sampleholograms and the color image sensor. A pixel super-resolved hologram ofthe sample is generated at each of the three or more distinct colorsusing the plurality of low resolution holographic images obtained bysimultaneous illumination of the sample by light from the three or moredistinct colors. De-multiplexed pixel super-resolved holograms are thengenerated at each of the three or more distinct colors using the pixelsuper-resolved hologram resulting from the simultaneous multi-colorillumination. Phase information is then retrieved from thede-multiplexed holograms at each of the three or more distinct colorsusing a phase retrieval algorithm to obtain a complex hologramcorresponding at each of the three or more distinct colors. The complexhologram for the three or more distinct colors is digitallyback-propagated and reconstructed to a sample plane to generate thecolor image of the sample by combining the reconstruction results ofeach of the three or more distinct colors.

In another embodiment, a system for generating color images of a sampleincludes an optically transparent sample holder configured to hold thesample thereon; one or more light sources configured to simultaneouslyoutput at least three different colors at a distance z₁ from the sampleon a first side of the sample holder; a color image sensor having acolor filter array (CFA), the color image sensor disposed on a secondside of the sample holder and having an active surface thereof locatedat a distance z₂ from the sample, wherein z₂ is significantly smallerthan z₁ (i.e., z₂«than z₁); and one or more processors configured toexecute image processing software thereon.

The image processing software obtains a plurality of low resolutionholographic images of the sample using the color image sensor, whereinthe simultaneously illuminated sample casts sample holograms on thecolor image sensor and wherein the plurality of low resolutionholographic images are obtained by relative x, y, and z directionalshifts between sample holograms and the color image sensor. The softwaregenerates a pixel super-resolved hologram of the sample using theplurality of low resolution holographic images obtained by simultaneousillumination followed by generating de-multiplexed pixel super-resolvedholograms at each of the at least three different colors using the pixelsuper-resolved hologram obtained from the multi-color, simultaneousillumination. Phase information is then retrieved from thede-multiplexed holograms at each of the at least three different colorsusing a phase retrieval algorithm to obtain a complex hologram at eachof the three or more distinct colors. Finally, the image processingsoftware digitally back-propagates and reconstructs the complex hologramfor each of the at least three different colors to a sample plane togenerate the color image of the sample by combining the reconstructionresults of each of the three or more distinct colors.

In one embodiment, the microscope imaging system is configured as abenchtop or desktop device. In another embodiment, the microscopeimaging system is configured as a hand-held or portable device that usesa modular attachment in conjunction with a portable electronic devicesuch as a mobile phone (e.g., Smartphone), tablet computer, webcam,laptop, or the like.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates one embodiment of a lens-free microscope system thatis used to simultaneously illuminate a sample with a plurality ofdifferent wavelengths (e.g., colors); whereby a plurality of lens-freecolor images are captured by a color image sensor having a CFA. Pixelsuper-resolved images are de-multiplexed and computationallyreconstructed to generate the final color image of the sample.

FIG. 1B illustrates a sample holder or substrate that contains a sampledisposed thereon. The sample holder may be formed from an opticallytransparent substrate (e.g., glass or plastic).

FIG. 1C illustrates an alternative lens-free microscope system to thebench or desktop embodiment of FIG. 1A. This embodiment uses a modularattachment that can be secured to a portable electronic device such as amobile phone (e.g., Smartphone). Images are acquired using the nativecolor image sensor of the mobile phone.

FIG. 1D illustrates the embodiment of FIG. 1C with the modularattachment being secured to the mobile phone.

FIG. 2 illustrates an example of a CFA (e.g., Bayer) that is used inconjunction with a color image sensor.

FIGS. 3A and 3B illustrate one method used to reconstruct phase andamplitude images of a sample according to one embodiment.

FIG. 4 illustrates a flowchart or diagram illustrating the flow ofoperations that are used to generate the final color image of a sampleusing simultaneous illumination according to one embodiment.

FIG. 5 illustrates another flowchart or diagram illustrating the flow ofoperations that are used to generate the final color image of a sampleusing simultaneous illumination.

FIG. 6A illustrates measured spectra of B, G1, G2 (G1 and G2substantially overlap) and R channels of the Bayer color image sensor,showing ˜15% cross-talk at the multiplexed wavelengths in theexperiments described herein (also see Table 1).

FIGS. 6B and 6C show reconstruction of a resolution test chart usingwavelength-multiplexed illumination with pixel super-resolution withoutthe de-multiplexing step. Significant distortions in high-resolutionfeatures are observed, as also shown with highlighted cross-sections.

FIGS. 6D and 6E illustrate reconstruction of the same data set of FIGS.6B and 6C with digital de-multiplexing (D-PSR). Previously unresolvablespatial features can now be resolved.

FIGS. 7A-7H illustrate the impact of saturation correction in D-PSRimages. Two different fields of view are illustrated in the upper panelof images (FIGS. 7A-7D) and the lower panel of images (FIGS. 7E-7H).FIGS. 7A and 7E illustrate D-PSR based image reconstruction withoutsaturation correction. FIGS. 7B and 7F illustrate D-PSR based imagereconstruction with the saturation correction step. FIGS. 7C and 7Gillustrate the same regions of interest reconstructed using sequentialRGB illumination (prior art), with a 3-fold increased number of rawholograms. FIGS. 7D and 7H illustrate the same samples imaged using alens-based microscope (40×, 0.75NA). Examples of prominent colorartifacts are marked with arrows in FIGS. 7A and 7E.

FIG. 8 illustrates a flow chart for the saturation correction steps oroperations used in the D-PSR approach according to one embodiment.

FIG. 9A illustrates the range of illumination wavelength combinationswith a maximum de-multiplexing error of 6% (shown as the outer surface)spans more than ˜50 nm for all three color channels. A typical selectionof red (˜610-650 nm), green (˜520-560 nm) and blue (˜450-480 nm)illumination wavelengths (shown with the inside cube) falls inside the6% maximum error volume.

FIG. 9B illustrates one-dimensional (1D) cross sectional plots of themaximum de-multiplexing error, each of which passes through the point(λ_(B), λ_(G), λ_(R))=(471,532,633)nm in FIG. 9A. The horizontal lineindicates an error threshold of 6%.

FIG. 10A compares a conventional demosaicing process (upper branch) thatgenerates color artifacts at holographic fringes, which are avoided inD-PSR (lower branch).

FIG. 10B illustrates a reconstructed image of a stained Pap smear sampleusing multi-height phase retrieval based hologram reconstruction fromfour heights using conventional demosaicing. A single hologram iscaptured under multi-wavelength illumination at each height.

FIG. 10C illustrates a reconstructed image of a stained Pap smear sampleusing multi-height phase retrieval based hologram reconstruction fromfour heights using D-PSR. 6×6 sub-pixel shifted holograms are capturedunder multi-wavelength illumination at each height.

FIG. 10D is a microscope image of the same regions of interest fromFIGS. 10A and 10B using a 40×, 0.75 NA objective lens.

FIG. 10E illustrates a reconstructed image of a stained breast cancertissue sample using multi-height phase retrieval based hologramreconstruction from four heights using conventional demosaicing. Asingle hologram is captured under multi-wavelength illumination at eachheight.

FIG. 10F illustrates a reconstructed image of a stained breast cancertissue sample using multi-height phase retrieval based hologramreconstruction from four heights using D-PSR. 6×6 sub-pixel shiftedholograms are captured under multi-wavelength illumination at eachheight.

FIG. 10G is a microscope image of the same regions of interest fromFIGS. 10E and 10F using a 40×, 0.75 NA objective-lens.

FIG. 11A illustrates a full field-of-view lens-free holographic imagethat is reconstructed using D-PSR under wavelength-multiplexedillumination at 470 nm, 527 nm and 624 nm. N=144 raw holograms are usedfor this reconstruction.

FIG. 11B illustrate a magnified region of FIG. 11A.

FIG. 11C illustrates the same magnified region-of-interest of FIG. 11Breconstructed using the YUV color-space averaging method. N+3=147 rawholograms are used. YUV color-space averaging method shows intensitybias and color leakage artifacts.

FIG. 11D illustrates the same magnified region-of-interest of FIG. 11Breconstructed using sequential RGB illumination. 3N=432 raw hologramsare used.

FIG. 11E illustrates the same magnified region-of-interest of FIG. 11Bobtained using a lens-based microscope. These microscope images areblurred in some regions due to limited depth-of-focus of theobjective-lens compared to lens-free holographic imaging. Typical FOVsof a 40× and a 20× objective-lens are also shown in FIG. 11A.

FIG. 11F illustrate a different magnified region of FIG. 11A.

FIG. 11G illustrates the same magnified region-of-interest of FIG. 11Freconstructed using the YUV color-space averaging method.

FIG. 11H illustrates the same magnified region-of-interest of FIG. 11Freconstructed using sequential RGB illumination.

FIG. 11I illustrates the same magnified region-of-interest of FIG. 11F

FIG. 11J illustrate a different magnified region of FIG. 11A.

FIG. 11K illustrates the same magnified region-of-interest of FIG. 11Jreconstructed using the YUV color-space averaging method.

FIG. 11L illustrates the same magnified region-of-interest of FIG. 11Jreconstructed using sequential RGB illumination.

FIG. 11M illustrates the same magnified region-of-interest of FIG. 11J.

FIG. 12 illustrates a flow-chart of de-multiplexing error calculationfor different combinations of multiplexed illumination wavelengths inD-PSR.

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

FIG. 1A illustrates one embodiment of a microscope system 10 that isused to generate color images of a sample 12. The sample 12 may include,for example, a tissue sample such as a thinly sliced tissue preparationthat is commonly used in histological and pathology applications. Forexample, the sample 12 may include a histology or pathology slide thatcontains tissue that is fixed and stained prior to visualization.Certain stains, for example, can be used so that certain cellularorganelles or features can be highlighted (e.g., cell nuclei as oneexample). In this embodiment, the microscope system 10 is a lens-freemicroscope device that includes a broadband light source 14 thatincludes a filter 16, such as an acousto-optic tunable filter that iscoupled to an optical fiber 18 (e.g., single mode fiber). The broadbandlight source 14 in conjunction with the filter 16 is able tosimultaneously output a plurality of different colors or wavelengths oflight. In one particular preferred embodiment of the invention, three(3) different colors or channels of light are emitted by the opticalfiber 18 and onto the sample 12. In one preferred embodiment, the three(3) different colors include red (R) light, green (G) light, and blue(B) light. It should be understood that the colors red, green and blueeach span a range of wavelengths. For example, red light is light thathas a wavelength within the range of about 610 nm to about 750 nm. Greenlight is light that has a wavelength within the range of about 495 nm toabout 570 nm. Blue light is light that has a wavelength within the rangeof about 450 to about 495 nm.

Thus, as used herein, the concepts of “red,” “green,” or “blue” light orred, green, or blue channels means that the light has a wavelengthgenerally within or close to the above-noted ranges. In some instances,the red, green, or blue light may include a single or narrow spectralband that spans one to a few nm. In other instances, the red, green, orblue light may span a larger range; yet still fall within the wavelengthranges described above.

As an alternative to a single light source 14 that is used to generate aplurality of different colors or wavelengths of light, multiple lightsources (e.g., light sources 14R, 14B, 14G as seen in FIG. 1A) may beused to simultaneous illuminate the sample 12. In this example, lightsource 14R emits red colored light while light source 14B emits bluecolored light and light source 14G emits green colored light. In thisparticular alternative embodiment, the different light sources 14R, 14B,14G may be formed using multiple light emitting diodes (LEDs) or laserdiodes. These light sources 14R, 14B, 14B are coupled to conventionaldriving circuitry (not shown) to simultaneously illuminate the sample12. Regardless of the form of the light source 14, the light ispartially coherent light that illuminates the sample 12.

The sample 12 is located on the sample holder 20. The sample holder 20is an optically transparent substrate such as glass or plastic that isused to hold a sample 12. For example, the sample holder 20 may includea glass slide or glass slip that is typically used to hold histologicalor pathological samples. The sample 12 that is contained on the sampleholder 20 includes objects 22 therein that are imaged by the lens-freemicroscope system 10. The lens-free microscope 10 is free of imagingforming units such as lenses, lens sets, lens modules, mirrors, orcombinations of the same. These objects 22 may include sub-cellularlevel objects or features (e.g., nuclei, organelles, and the like). Ofcourse, the sample 12 may also include a sample of non-biologicalorigin.

The lens-free microscope system 10 includes a color image sensor 24 thatis located adjacent to the underside of the sample holder 20. The colorimage sensor 24 may be CMOS-based and includes an array of pixels 26 asseen in FIG. 2. In some instances, the underside of the sample holder 20may actually be in contact with the color image sensor 24. The colorimage sensor 24 includes a color filter array (CFA) 28. The particularCFA that is used by the image sensor 24 may vary. Examples include aBayer filter, RGBE filter, CYYM filter, CYGM filter, RGBW Bayer filter,and RGBW filter. FIG. 2 illustrates a color image sensor 24 thatincludes a Bayer CFA. As seen in FIG. 2, each pixel 26 of this colorimage sensor 24, in response to the Bayer CFA 28 has four channels,namely B (Blue), G1 (Green 1), G2 (Green 2), and R (Red), which form a2×2 period of the Bayer pattern on the color image sensor 24. Eachchannel of the Bayer CFA 28 operates as a filter to permit thetransmission of a different range of wavelengths. Each channel of theCFA 28 has different transmission spectra.

The distance between the output of the partially coherent light source14 (or multiple sources 14R, 14G, 14B) and the sample 12 referred to asthe z₁ distance is generally on the order of several centimeters (e.g.,˜5-15 cm). The active surface (i.e., imaging surface) of the color imagesensor 24 is located a distance z₂ below the sample 12 and issignificantly smaller as compared to the z₁ distance (i.e., z₂«z₁). Thetypical distance for the z₂ dimension is generally less than 1 mm and,in other embodiments, between about 100 μm to about 600 μm. The colorimage sensor 24 in the lens-free microscope system 10 is used to captureholographic images of objects 22 and features contained in the sample12. Objects 22 may include sub-cellular features such as organelles orthe like (e.g., nuclei) that are present in the sample 12. Objects 22may also include non-biological objects such as beads, etc.

With reference to FIG. 1A, the lens-free microscope system 10 furtherincludes, in one embodiment, a translation stage 30 that, in oneembodiment, is coupled to the color image sensor 24 and moves the colorimage sensor 24 in the x or y directions which lie in a plane that issubstantially parallel with the active surface of the color image sensor24 or in the z direction which, as illustrated, is generally orthogonalto the plane of the active surface of the color image sensor 24.Movement in the x or y directions is used to capture images of thesample 12 using pixel super-resolution. In order to generatesuper-resolved images, a plurality of different, lower resolution imagesare taken as color image sensor 24 is moved in small increments in the xand y directions. In another alternative embodiment, the optical fiber18 is moved in small increments generally in the x and y directions bythe translation stage 30. In yet another alternative, the sample holder20 may be moved in small increments in the x and y directions. Thetranslation stage 30 may, optionally, be automatically controlled usinga computer 32, dedicated controller, or the like to control an actuatingelement. Manual control of the translation stage 30 is also an option.Any number of mechanical actuators may be used including, for example, astepper motor, moveable stage, piezoelectric element, or solenoid. Thetranslation stage 30 may also be manually-operated stage. Preferably,the translation stage 30 can move in sub-micron increments therebypermitting images to be taken of the sample 12 at slight x and ydisplacements.

In still another alternative embodiment, rather than move the opticalfiber 18 in the x and y directions, a plurality of spaced apartillumination sources (e.g., an array of light sources 14 not shown) canbe selectively actuated to achieve the same result without having tophysically move the optical fiber 18 or color image sensor 24. The smalldiscrete shifts (either by movement or actuation of spatially separatedlight sources 14) parallel to the color image sensor 24 are used togenerate a pixel super-resolution hologram image that includes thechannels of the CFA 28. For a Bayer CFA 28 which four channels, thepixel super-resolution hologram includes the four Bayer channels of B(Blue), G1 (Green 1), G2 (Green 2), and R (Red). In addition to movementin the x and y directions, the translation stage 30 may also move thesample holder 20 and/or color image sensor 24 in the z direction (i.e.,orthogonal to x, y plane) so that images may be obtain at multipleheights. This enables multi-height phase recovery as described in moredetail below.

FIGS. 1C and 1D illustrate an alternative embodiment of the microscopesystem 10′. In this embodiment, rather than have a benchtop or desktoplens-free microscope system 10 as illustrated in FIG. 1A, the samefunctionality may be incorporated into a lens-free based modularattachment 50 that is used in conjunction with a portable electronicdevice 52 such as a mobile phone. Other portable electronic devices 52include tablets computing devices, webcams, laptops, and the like. Themobile phone 52 may include, for example, a Smartphone. Any number ofmakes and models of the mobile phone 52 may be used with the system 10′and methods described herein. The mobile phone 52 includes housing 54that contains a color image sensor 56 (FIG. 1C) that is used to providecamera functionality for the mobile phone 52. The color image sensor 56,like color image sensor 24 in FIG. 1A, includes a CFA 28. The mobilephone 52 further includes an internal lens 58 (FIG. 1C) that is disposedwithin the housing 54 of the mobile phone 52.

As seen in FIGS. 1C and 1D, the lens-free modular attachment 50 includesa plurality of partially coherent light sources 60 which emit light atdifferent colors or wavelengths. Illustrated in FIGS. 1C and 1D includesa red (R) light source 60, a green (G) light source 60, and a blue (B)light source 60. The light sources 60 may include one or more LEDs orlaser diodes. The modular attachment 50 includes an aperture 62 or itsequivalent for spatial filtering, and a slot 64, tray, receptacle or thelike that can receive the sample holder 20 containing the sample 12 soas to place the sample 12 in an optical path formed between the lightsources 60 and the color image sensor 56. The lens-free modularattachment 50 also includes a translation stage 66 which allows formovements in the x, y, and z directions to obtain a pixelsuper-resolution image as described herein. Knobs 68 may be used to movethe sample holder 20 in the x and y directions while knob 70 may be usedto move the sample holder 20 in the z direction. Images may betransferred to a computer 32 such as that illustrated in FIG. 1A using awired or wireless connection. For example, the mobile phone 52 maycontain software or an application (i.e., “app”) that is used to acquirethe lower resolution lens-free images which can then be offloaded ortransferred to the computer 32 for further processing. The lens-freebased modular attachment 50 may also include mechanical grips, tabs,clips 72 or the like to secure the same to the phone as seen in FIG. 1D.

In the pixel super-resolution image process, a plurality of lowerresolution images are taken at different positions and are used togenerate a computational image reconstruction that has high resolution.As seen in FIG. 3A, in step 1000, a plurality of lower resolution imagesare obtained of the sample 12 while the illumination source(s) 14,sample holder 20, and/or the color image sensor 24 are moved relative toanother at a plurality of different locations (e.g., x, y locations) tocreate the sub-pixel image shifts. The number of lower resolution imagesmay vary but generally includes between about 2 and 250 images. Duringstep 1000, the sample 12 is located from the image sensor 24 at a firstdistance (d₁). Next, as seen in step 1100, a pixel super-resolved (PSR)hologram is synthesized based upon the plurality of lower resolutionimages obtained in operation 1000. The details of digitally converting aplurality of lower resolution images into a single, higher resolutionpixel super-resolved hologram image may be found in Bishara et al.,Lensfree on-chip microscopy over a wide field-of-view using pixelsuper-resolution, Optics Express 18:11181-11191 (2010) and Greenbaum etal., Imaging without lenses: achievements and remaining challenges ofwide-field on-chip microscopy, Nat. Methods 9, 889-895 (2012), which areincorporated herein by reference. This pixel super-resolution step takeslower resolution holographic shadows of the object(s) 22 containedwithin the sample 12 (e.g., captured at ˜10 million pixels each) andthen creates a higher resolution lens-free hologram that nowcontains>300 million pixels over the same 30 mm² field-of-view with aneffective pixel size of ˜300 nm.

Next, as seen in operation 1200, the distance between the sample 12 andthe color image sensor 24 is adjusted to a different distance (d_(n))(e.g., by adjusting z distance using translation stage 30). At this newdistance (d_(n)), as seen in operation 1300, a plurality of lowerresolution images are obtained of the sample 12 containing the object(s)22 while the illumination source(s) 14, sample holder 20, and/or thecolor image sensor 24 are moved relative to another at a plurality ofdifferent locations (e.g., x, y locations) to create the sub-pixel imageshifts. The plurality of lower resolution hologram images are obtainedwhile the sample 12 and the color image sensor 24 are located at the newor different distance (d_(n)). After the lower resolution images areobtained, as seen in operation 1400, a pixel super-resolved hologram (atthe different distance (d_(n))) is synthesized based upon the pluralityof lower resolution images obtained in operation 1300. As seen by arrow1500, process is repeated for different sample-to-sensor differences.Generally, the process repeats such that a pixel super-resolved hologramis created at between 2-20 different distances although this number mayvary. For example, in experiments described herein, four (4) suchheights were used for the D-PSR method. Alternatively, the lowerresolution images are all obtained at different distances and then foreach distance, the higher resolution pixel super-resolved holograms arethen recovered.

Now referring to FIG. 3B, the plurality of pixel super-resolvedholograms obtained at the different heights (i.e., different zdistances) are then registered with respect to each other as seen inoperation 1600. The subsequent iterative phase recovery requires thatthese pixel super-resolved holograms are accurately registered to eachother. During the image acquisition step, lateral translation androtation of the objects between holograms of different heights areunavoidable. To accurately register these pixel super-resolved hologramsto each other, three-control points from three different corners of theimage are selected in one of the holograms (which is deemed thereference hologram). One preferred control point could be a smallisolated dust particle at a corner since its hologram is circularlysymmetric. If need be, a special alignment marker(s) can also be placedat the corners of the sample holder/substrate. Therefore, normalizedcorrelations between lens-free holograms can be used to find thematching points in each image captured at a different height. Afterselection of the control points, a small area (e.g., ˜30×30 μm) aroundeach control point is cropped and digitally interpolated (˜4-6 times) toserve as a normalized correlation template. Furthermore, for accuratelyfinding the coordinate shift of each control point among M images,lens-free holographic images have to be positioned in the samez₂-distance. Therefore, the difference in the z₂-distance betweenlens-free holograms acquired at different heights is evaluated by anauto-focus algorithm, such as that disclosed in J. L. Pech-Pacheco etal., “Diatom Autofocusing in Brightfield Microscopy: a ComparativeStudy,” in Pattern Recognition, International Conference On (IEEEComputer Society, 2000), Vol. 3, p. 3318, incorporated herein byreference, which permits one to digitally propagate the selectedcorrelation templates to the same z₂-distance, where normalizedcorrelations are calculated to find the coordinate shifts between thecontrol points in each image. An affine transformation is used toregister the super-resolved holograms of different heights to thereference hologram.

Still referring to FIG. 3B, operations 1700, 1800, 1900, and 2000illustrate one embodiment of the iterative phase recovery process thatis used to recover the lost optical phase. Additional details regardingthe iterative phase recovery process may be found in L. J. Allen and M.P. Oxley, Optics Communications, 2001, 199, 65-75, which is incorporatedherein by reference. The square roots of these resulting M registeredholograms are then used as amplitude constraints in the iterative phaserecovery algorithm that is steps 1700 through 2000. At the beginning ofthe algorithm, as seen in operation 1700, in one embodiment, the initialphase is assumed to be zero, after which the iterative phase recoveryalgorithm uses the free space propagation function to digitallypropagate back and forth among these multiple heights. At each height,the amplitude constraint (i.e., the measurement) is enforced while thephase is kept from the previous digital propagation step.

To initiate the phase recovery process, a zero-phase is assigned to theobject intensity measurement. One iteration during this phase-recoveryprocess can be described as follows: Intensity measurement #1 (step1700) is forward propagated (with zero initial phase) to the plane ofintensity measurement #2 (step 1800). Then, the amplitude constraint inmeasurement #2 (step 1800) is enforced while the calculated phaseresulting from forward propagation remains unchanged. The resultingcomplex field is then forward propagated to the plane of intensitymeasurement #3 (step 1900), where once again the amplitude constraint inmeasurement #3 is enforced while the calculated phase resulting fromforward propagation remains unchanged. This process continues untilreaching the plane of intensity measurement #M (step 2000). Then insteadof forward propagating the fields of the previous stages, backpropagation is used as seen by respective arrows A, B, and C. Thecomplex field of plane #M (step 2000) is back propagated to the plane ofintensity measurement #M−1. Then, the amplitude constraint inmeasurement #M−1 is enforced while the resulting phase remainsunchanged. The same iteration continues until one reaches the plane ofintensity measurement #1 (step 1700). When one complete iteration isachieved (by reaching back to the plane of intensity measurement #1),the complex field that is derived in the last step will serve as theinput to the next iteration. Typically, between 1-1,000 iterations andmore typically between 1-70 iterations are required for satisfactoryresults (more typically between 20-30 iterations). After the phaserecovery iterations are complete, as seen in operation 2100, theacquired complex field of any one of the measurement planes is selectedand is back propagated to the object plane to retrieve both phase image2200 and amplitude image 2300 of the sample 12.

As explained further herein, multi-height phase recovery may beincorporated by utilizing the solution to the transport of intensityequation (TIE) to obtain the initial guess of the lost phase as well astilt correction. Details regarding the use of TIE to generate theinitial phase guess to multi-height based iterative phase retrieval aswell as tilt correction may be found in U.S. Patent ApplicationPublication No. 2017-0220000, which is incorporated by reference herein.

FIGS. 4 and 5 illustrate a flowchart or diagram illustrating the flow ofoperations according to one embodiment. With reference to FIG. 4, theprocess starts in operation 3000 where the sample 12 is illuminatedsimultaneously by the illumination source(s) 14, 60 with multiple colorsor wavelengths. The color image sensor 24 is used to acquire hologramimages with sub-pixel x, y shifts at different heights as seen byoperation 3100 in FIG. 4. FIG. 5 illustrates, for example, nine (9)sub-pixel shifted frames that have been captured by the illuminationsource(s) 14 at a single height (different numbers may be used). Thesesub-pixel images are obtained at a plurality of different heights.Referring back to FIG. 4, in operation 3200, a pixel super-resolvedhologram is generated for each color channel of the CFA for each height.FIG. 5, for example, illustrates a four channel image (R, G1, G2, B)being generated for a Bayer CFA 28. Next, as seen in operation 3300 ofFIG. 4, image de-multiplexing of the pixel super-resolved holograms isperformed using the spectral cross-talk matrix (W) for each height. FIG.5 illustrates the four channel (R, G1, G2, B) pixel super-resolutionimage that is de-multiplexed into three channels (R, G, B). Referringback to FIG. 4, next in operation 3400, a multi-height phase retrievalprocess is performed to digitally reconstruct complex projection imagesof the sample with the iterative phase recovery process. This process isalso illustrated in FIG. 5 with the TIE being used for the initialguess. As seen in FIG. 4, in operation 3500, a recovered image is thenback propagated to the object or sample plane to create phase andamplitude images of the sample. FIG. 5 illustrates the recovered imagesfor the R, G, and B channels. These different channels can then becombined in operation 3600 (FIG. 5) with white balancing to produce thefinal demosaiced pixel super-resolution (D-PSR) color image as seen inFIG. 5.

Referring back to FIG. 1A, the microscope system 10 includes a computer32 such as a server, laptop, desktop, tablet computer, portablecommunication device (e.g., Smartphone), personal digital assistant(PDA) or the like that is operatively connected to the microscope system10 such that lower resolution images (e.g., lower resolution or rawimage frames) are transferred from the color image sensor 24 to thecomputer 32 for data acquisition and image processing. The computer 32includes one or more processors 34 that, as described herein in moredetail, runs or executes image processing software 36 that takesmultiple, sub-pixel (low resolution) images taken at different scanpositions (e.g., x and y positions as seen in inset of FIG. 1A) andcreates a high resolution projection hologram image of the objects 22 inthe sample 12 for each color channel (e.g., R, G1, G2, B). The software36 creates additional high resolution projection hologram images of theobjects 22 at each different z₂ distance. The multiple, high resolutionimage “stacks” that are obtained at different heights are registeredwith respect to one another using the software 36.

As explained herein, the software 36 takes the multi-channel color pixelsuper-resolution images and then subjects the images to imagede-multiplexing. In image de-multiplexing, the transmission spectra ofthe CFA 28 typically has considerable cross-talk among the colorchannels. For example, for a Bayer CFA 28, for each pixel of the colorimage sensor 24, one can formulate this spectral cross-talk as a matrix(W), such that:

$\begin{matrix}{\begin{bmatrix}M_{B} \\M_{G1} \\M_{G2} \\M_{R}\end{bmatrix} = {{W \cdot \begin{bmatrix}1_{B} \\1_{G} \\1_{R}\end{bmatrix}} = {\begin{bmatrix}w_{11} & w_{12} & w_{13} \\w_{21} & w_{22} & w_{23} \\w_{31} & w_{32} & w_{33} \\w_{41} & w_{42} & w_{43}\end{bmatrix} \cdot \begin{bmatrix}1_{B} \\1_{G} \\1_{R}\end{bmatrix}}}} & (1)\end{matrix}$

where M_(B), M_(G1), M_(G2), and M_(R) correspond to the pixelsuper-resolved intensity values for each channel (i.e., the output ofthe previous sub-section), and I_(B), I_(G), and I_(R) refer to thede-multiplexed holograms corresponding to the three illuminationwavelengths, before the spectral mixing occurred at the color imagesensor 24. The entries of the cross-talk matrix W are determined by thetransmission spectra of the Bayer CFA 28. These may be provided by themanufacturer of the color image sensor 24 or experimentally determined.Importantly, for a given color image sensor 24, the spectral cross-talkcalibration curves need only be measured once.

Based on the spectral cross-talk matrix, the de-multiplexed hologramscorresponding to the three simultaneous illumination wavelengths (i.e.,R, G, B) in the microscope system 10 can then be determined through aleft inverse operation:

$\begin{matrix}{\begin{bmatrix}I_{B} \\I_{G} \\I_{R}\end{bmatrix} \approx {\left( {W^{T}W} \right)^{- 1}{W^{T}\begin{bmatrix}M_{B} \\M_{G1} \\M_{G2} \\M_{R}\end{bmatrix}}}} & (2)\end{matrix}$

where the superscript −1 refers to the inverse and T refers to thetranspose of a matrix. Post de-multiplexing, as seen in FIGS. 4 and 5,there exists de-multiplexed, pixel super-resolved images for the threecolor channels R, G, B at a given height. A plurality of thesede-multiplexed images is generated for the multiple heights (i.e., z₂distances) where images have been obtained. The software 36 thendigitally reconstructs complex projection images of the sample 12 and/orobjects 22 through an iterative phase recovery process that rapidlymerges all the captured holographic information to recover lost opticalphase of each lens-free hologram without the need for any spatialmasking, filtering, or prior assumptions regarding the samples. After anumber of iterations (typically between 1 and 75), the phase of eachlens-free hologram (captured at different heights) is recovered and oneof the pixel super-resolved holograms is back propagated to the objectplane to create phase and amplitude images of the sample 12 includingobjects 22 contained therein.

The computer 32 may be associated with or contain a display 38 or thelike that can be used to display color images that are generated inaccordance with the methods described herein. The user may, for example,interface with the computer 32 via an input device 40 such as a keyboardor mouse to select different software functions using a graphical userinterface (GUI) or the like. It should be noted that the methoddescribed herein may also be executed in a cloud-based processingoperations. Image data could be sent to a remote computer 32 (e.g.,remote server) for processing with a final image being generatedremotely and sent back to the user on a separate computer 32 or otherelectronic device (e.g., mobile phone display) for ultimate display andviewing. Image and other data may be transferred over a wide areanetwork such as the Internet or a proprietary communication network(like those used for mobile devices).

Experimental

Experiments were performed to demonstrate that pixel super-resolutioncan be merged into the color de-multiplexing process to significantlysuppress the artifacts in wavelength-multiplexed holographic colorimaging where multiple wavelengths (e.g., three) simultaneouslyilluminate a sample. This new D-PSR approach generates color images thatare similar in performance to sequential illumination at threewavelengths, and therefore improves the speed of holographic colorimaging by 3-fold. D-PSR method is broadly applicable to holographicmicroscopy applications, where high-resolution imaging andmulti-wavelength illumination are desired.

Optical Setup and Data Acquisition

With reference to FIGS. 1A and 1B, an in-line holographic lens-freeon-chip imaging geometry was used. A broadband source (WhiteLase-Micro;Fianium Ltd, Southampton, UK) is filtered by an acousto-optic tunablefilter down to ˜5 nm bandwidth and is coupled to a single mode fiber togenerate partially-coherent illumination across the field-of-view (˜20mm²) of the lens-free imaging setup. This source can simultaneouslyoutput up to eight (8) wavelength channels into the same fiber opticcable, and three spectral bands were used in the experiments (i.e., ˜470nm, ˜530 nm, ˜625-630 nm) to create multi-color illumination. Thismultiplexed partially coherent light, coming out of the fiber opticcable, propagates ˜6 cm, and impinges on the specimen plane. Note thatin other commercial or practical embodiments, the light source mayinclude multiple different light sources as explained herein that emitlight at a particular wavelength or narrow band of wavelengths. Thesources may include, for example, LEDs or laser diodes. With respect tothe experimental results generated herein, each of the three wavelengthchannels is partially diffracted by the sample and generates threeindependent in-line holograms to be sampled by a Bayer color CMOS imagesensor chip (16.4 Mpixel, 1.12 μm pixel size, Sony Corp., Japan), whichis placed ˜0.4 mm below the sample plane. The pixels of this color CMOSimage sensor chip have four channels, namely B (Blue), G1 (Green 1), G2(Green 2), and R (Red), which form a 2×2 period of the Bayer pattern onthe image sensor chip (see FIG. 2). These filters have differenttransmission spectra, which will be detailed later on, and thisinformation is crucial for spectral de-multiplexing of the acquiredholograms. The color CMOS image sensor chip is also mounted on acomputer controlled 3D motion stage to permit: (1) lateral sub-pixelshifts between the color image sensor and the sample hologram (x, ydirection) which is used to generate pixel super-resolved holograms, and(2) axial modulation (z direction) of the sample-to-sensor distancewhich is used for multi-height based phase retrieval (i.e., twin-imageelimination). The entire data acquisition process is automated by acustom-developed LabVIEW program.

Pixel Super-Resolution

Pixel super-resolution is a technique that deals with the spatialunder-sampling problem in an imaging system, in which a series ofsub-pixel shifted low resolution images are acquired to digitallysynthesize a high resolution image of the object, significantlyincreasing the space-bandwidth product of the imaging system. In theseexperiments, to achieve pixel super-resolution, the stage was programmedto move the image sensor laterally on a 6×6 grid and at each grid pointa low-resolution raw hologram is captured. Each recorded raw hologramintensity is then separated into four Bayer channels (namely B, G1, G2,and R) and for each one of these channels, a conjugate gradient basedpixel super-resolution method was used to synthesize a super-resolvedhologram with an effective pixel size of ˜0.33 μm at eachsample-to-sensor height. The spatial location of each channel withrespect to the others is also taken into account and digitally correctedfor; therefore this pixel-super resolution step enables all the Bayerchannels (B, G1, G2 and R) to be virtually super-imposed onto eachother, which is important to mitigate the artifacts in the subsequentdemosaicing steps.

De-Multiplexing of Pixel Super-Resolved Holograms

The transmission spectra of the four Bayer channels on a color CMOSimage sensor contain considerable color cross-talk among channels (seeFIG. 6A). For each pixel of the image sensor chip, this spectralcross-talk can be formulated as a matrix (W), as described by Equation 1herein. The entries of the cross-talk matrix W are determined by thetransmission spectra of the Bayer CFA. Although the transmissionspectrum of each Bayer filter is usually provided by the manufacturer ofthe sensor-array, here it was experimentally calibrated to get moreaccurate results. For this purpose, the background (i.e., object-free)response of the color image sensor was first recorded from 400 nm to 700nm at 5 nm steps.

A 400-by-400 pixel region at the center of the sensor chip was averagedfor each channel, and after normalization of the illumination power ateach wavelength, measured using a power-meter (Thorlabs PM100, S120UVsensor head), the resulting curve for each channel is then taken as thespectral response of each Bayer filter on the image sensor chip (seee.g., FIG. 6A). It should be emphasized that for a given color imagesensor, these spectral cross-talk calibration curves need to be measuredonly once. Based on these measured spectra, the cross-talk matrix (W inEq. (1)) can be inferred for any arbitrary set/choice of illuminationwavelengths that are multiplexed in the holographic color imagingexperiments (see e.g., Table 1).

TABLE 1 Calibrated cross-talk matrix of the CMOS image sensor chip (SonyIMX81) at two sets of multiplexed wavelengths. λ_(B), λ_(G), λ_(R) 470nm, 527 nm, 624 nm 471 nm, 532 nm, 633 nm W $\quad\begin{bmatrix}1.0000 & 0.1722 & 0.0566 \\0.2903 & 1.0000 & 0.2174 \\0.2873 & 1.0003 & 0.2030 \\0.0307 & 0.1349 & 1.0000\end{bmatrix}$ $\quad\begin{bmatrix}1.0000 & 0.1593 & 0.0650 \\0.3340 & 1.0000 & 0.2062 \\0.3353 & 1.0055 & 0.1936 \\0.0335 & 0.1345 & 1.0000\end{bmatrix}$

Based on this spectral cross-talk matrix, the de-multiplexed hologramscorresponding to the three simultaneous illumination wavelengths in theholographic imaging set-up can then be determined through a left inverseoperation using Equation 2 above.

Multi-Height Based Phase Retrieval

One drawback of in-line holographic imaging geometry is its twin imagenoise. Additional constraints, such as the object support, sparsity ormultiple measurements at different heights or illumination angles can beemployed to eliminate the twin image noise. For spatially dense andconnected objects a multi-height based phase retrieval method is usuallyused because it is relatively hard to define an object support for suchconnected samples. In this multi-height based iterative phase retrievalalgorithm, one starts from one of the pixel super-resolved hologram anddigitally propagates it to the next measurement height, where theamplitude of the field is replaced with the measured amplitude, and thenpropagates it to the next height until one reaches the last measurementplane (z). The same process is repeated backward and then forward fore.g., 20-30 iterations. Each wave propagation operation is done usingthe angular spectrum method. For faster convergence, optionally, one canuse the solution to the transport-of-intensity equation (TIE) as theinitial phase guess for multi-height phase retrieval. In the experimentsreported herein, holograms were measured at four (4) consecutive heightsthat are axially separated by ˜30 μm.

Saturation Correction in Digitization of Wavelength-MultiplexedHolograms

When using the D-PSR approach for imaging of biological samples, asaturation-related de-multiplexing color artifact can sometimes beobserved, as also illustrated in FIGS. 7A-7H. Although pixel saturationcan be avoided by reducing the exposure time to a point where no pixelsare saturated, this will result in unacceptable loss of information, asmost of the pixels will then use only a small portion of the dynamicrange. Alternatively, here a Bayesian-estimation-based saturationcorrection algorithm is used, which uses the unsaturated pixels fromother color channels at the same physical location to get a betterestimate of the saturated pixels. It is theoretically proven that, usingthis Bayesian estimation approach, the corrected image will always havea smaller error than the uncorrected saturated one.

Details regarding the use of this saturation correction method in theD-PSR approach are presented below. It is assumed that for a given rawimage, the pixel values of different color channels follow a normaldistribution:

$\begin{matrix}{\begin{pmatrix}X_{s} \\X_{k}\end{pmatrix} \sim {N\left\lbrack {\begin{pmatrix}\mu_{s} \\\mu_{k}\end{pmatrix},\begin{pmatrix}s_{ss} & s_{sk} \\s_{ks} & s_{kk}\end{pmatrix}} \right\rbrack}} & (3)\end{matrix}$

where X_(s) and X_(k) denote pixel values of saturated and unsaturatedchannels, μ_(s) and μ_(k) represent their mean, respectively, andS_(ss), S_(sk), S_(ks) and S_(kk) represent their covariance. Thesaturated channel X_(s) can be replaced by its statistical expectation,using the known non-saturated channel measurements X_(k)=k at the samepixel location:

$\begin{matrix}{{E\left( {{\left. X_{s} \middle| X_{k} \right. = k},\ {X_{s} \geq s}} \right)} = {\mu_{xs} + {\frac{1}{z}\left( \frac{S_{xs}}{2\pi} \right)^{\frac{1}{2}}{\exp\left\lbrack {- \frac{\left( {s - \mu_{xs}} \right)^{2}}{2S_{xs}}} \right\rbrack}}}} & (4)\end{matrix}$

where:

$\begin{matrix}{Z = {\frac{1}{\sqrt{2\;\pi\; S_{x\; s}}}{\int_{s - \mu_{xs}}^{\infty}{{\exp\left( {- \frac{x^{2}}{2\; S_{xs}}} \right)}{dx}}}}} & (5) \\{\mu_{xs} = {\mu_{s} + {S_{sk}{S_{kk}^{- 1}\left( {k - \mu_{k}} \right)}}}} & (6) \\{S_{xs} = {S_{ss} - {S_{sk}S_{kk}^{- 1}S_{sk}^{T}}}} & (7)\end{matrix}$

Note that since the spectral response of G1 and G2 channels are nearlyidentical, the average is taken of these two super-resolved channels andit is treated as the same channel G—only for this saturation correctionstep. The saturation correction algorithm (see FIG. 8) is then implantedin five steps, as follows:

Step 1. Estimate the a-priori mean {circumflex over (μ)} and co-varianceŜ of the unsaturated pixel values of R, G and B channels:

$\begin{matrix}{\hat{\mu} = {\begin{pmatrix}{\hat{\mu}}_{R} \\{\hat{\mu}}_{G} \\{\hat{\mu}}_{B}\end{pmatrix} = {\frac{1}{n} \cdot {\sum\limits_{j}^{n}x_{j}}}}} & (8) \\{\hat{S} = {\begin{pmatrix}{\hat{S}}_{RR} & {\hat{S}}_{RG} & {\hat{S}}_{RB} \\{\hat{S}}_{GR} & {\hat{S}}_{GG} & {\hat{S}}_{GB} \\{\hat{S}}_{BR} & {\hat{S}}_{BG} & {\hat{S}}_{BB}\end{pmatrix} = {\frac{1}{n - 1} \cdot {\sum\limits_{j}^{n}{\left( {x_{j} - \hat{\mu}} \right)\left( {x_{j} - \hat{\mu}} \right)^{T}}}}}} & (9)\end{matrix}$

where n is the total number of un-saturated pixels in the image,x_(j)=(x_(j) ^(R), x_(j) ^(G), x_(j) ^(B))^(T) is a vector thatrepresents the pixel values of R, G, B channels at pixel location j.

Step 2. After defining a saturation level s, the distance d_(i) of allthe channels (i=R, G, B) can be determined as:

$\begin{matrix}{{d_{i} = \frac{s - {\hat{\mu}}_{i}}{\sqrt{v_{i}}}},{{{for}\mspace{14mu} i} = R},G,B} & (10)\end{matrix}$

where {circumflex over (μ)}_(i) and v_(i) define the mean and thevariance of all the unsaturated pixels in color channel i, respectively.Here, s=1020 was chosen for the 10 bit depth image sensor.

Step 3. Start from the most saturated channel, i.e. the channel i (i=R,G or B) that has the smallest distance d_(i) to the saturation level,and replace the values of its saturated pixels with the expectationvalue calculated using Eq. (4). All the pixels in the other twoun-corrected channels are taken as valid pixels.

Step 4. Correct the second most saturated channel i using Eq. (4),taking the corrected most saturated channel and the other un-correctedchannel as valid pixels.

Step 5. Correct the third (last) saturated channel using Eq. (4), takingthe corrected values of the first and the second most saturated channelsas valid pixels.

Steps 3-5 are typically run iteratively (e.g., for 3 iterations) to getimproved results. As illustrated in FIGS. 7A-7H, the de-multiplexingcolor artifacts shown in the first column (FIGS. 7A and 7E) are greatlyalleviated with this additional saturation correction step (secondcolumn—FIGS. 7B and 7F), resulting in a reconstructed color image thatis similar to a sequentially taken RGB image (third column—FIGS. 7C and7G).

White-Balancing of Wavelength-Multiplexed Holograms

Although the power levels of the multiplexed illumination wavelengthsduring the measurements are adjusted so that their detected intensitiesare very close to each other, there are still small uncontrolledvariations among color channels. To correct for these power variations,a uniform background (empty) region of the captured hologram is firstchosen and then one calculates the average of each Bayer channel withinthis selected region which is taken as the relative power level of eachillumination wavelength. All the reconstructed holographic images arethen normalized using these calculated power ratios to get awhite-balanced image.

Optimization of the Choice of Illumination Wavelengths in D-PSR

Typically three illumination wavelengths are multiplexed in the D-PSRexperiments, which are assigned to B, G and R channels, respectively.Here, the following question was also addressed: if one couldarbitrarily choose these three illumination wavelengths, what would bethe optimal wavelength range for each source to be multiplexed?Intuitively, the optimality of the selected wavelengths depends on thetransmission spectra (i.e., wavelength cross-talk) of the color filterson the color image sensor chip, as well as the transmissioncharacteristics of the specimen to be imaged. Since the aim here is forgeneral purpose microscopic imaging, optimization of the illumination asa function of the sample spectral characteristics is not considered; andtherefore only considered the transmission spectra of the CFA on thecolor image sensor.

If the multiplexed channels are chosen to be too close in wavelength,the cross-talk among them will be too strong, and the illumination powerof one or more channels needs to be reduced to accommodate the finitebit-depth of the digital sensor, which in turn will cause loss ofspatial information. To better understand how this de-multiplexing errorvaries according to the selection of the multiplexed illuminationwavelengths, a brute-force search was conducted of all the possiblewavelength combinations for the spectral range of 400 nm to 700 nm with1 nm step size and the resulting de-multiplexing errors were compared.FIG. 9 illustrates a flow-chart of de-multiplexing error calculation fordifferent combinations of multiplexed illumination wavelengths in D-PSR.

FIG. 9A illustrates the range of illumination wavelength combinationswith a maximum de-multiplexing error of 6% (shown as the outer surface)spans more than ˜50 nm for all three color channels. A typical selectionof red (˜610-650 nm), green (˜520-560 nm) and blue (˜450-480 nm)illumination wavelengths (shown with the inner cube) falls inside the 6%maximum error volume. FIG. 9B illustrates 1D cross sectional plots ofthe maximum de-multiplexing error, each of which passes through thepoint (λ_(B), λ_(G), λ_(R))=(471,532,633)nm in FIG. 9A. The dashed blackline indicates an error threshold of 6%.

As illustrated in FIGS. 9A and 9B, the differences among thede-multiplexing errors for different wavelength combinations are smallerthan 6% over a large spectral range (>50 nm), which also contains thetypical choice of red (610-650 nm), green (520-560 nm) and blue (450-480nm) illumination bands. Based on this, it was concluded that for atypical Bayer image sensor, like the image sensor used herein, the rangeof wavelength combinations that can be used for simultaneousillumination of the sample is rather large.

Results and Discussion

When the illumination wavelengths are multiplexed and simultaneouslyrecorded, the resulting holograms using a Bayer image sensor chip, therewill be mainly two types of artifacts generated: (1) the spectralcross-talk among different Bayer filters will create pixel level mixingof holographic information of different illumination wavelengths (seee.g. FIG. 6A); and (2) spatial demosaicing artifacts will be createdbecause the Bayer mosaicing geometry has four (4) color channels (B, G1,G2, and R) that are spatially separated by one pixel shift, and requiresthe interpolation of neighboring pixels for the missing spatialinformation, which gives rise to fringe artifacts in holograms.Conventional demosaicing techniques employed in digital cameras andphotography literature rarely suffer from these artifacts as mostnatural images are spatially smooth. However, when dealing withmulti-color digital holographic microscopy, the recorded hologramscontain rapid oscillations and fringes, and therefore using aconventional demosaicing approach will result in severe color artifacts.

The first problem listed above, i.e., the spectral cross-talk issue, cangenerate strong high-frequency artifacts if left uncorrected.Experimental examples of these artifacts are illustrated in the imagesof FIGS. 6B and 6C, where pixel super-resolved holographic imagereconstructions are shown without de-multiplexing. These are to becompared against the de-multiplexed D-PSR results (FIGS. 6D and 6E),which show significant improvements especially in high-resolutionfeatures. As explained herein, this issue can be tackled by digitalde-multiplexing (through Eq. (2)). However, if this de-multiplexing stepis performed directly on demosaiced Bayer pixels (i.e., without pixelsuper-resolution), it will also generate color artifacts for holographicimaging at interpolated fringes (e.g., see FIG. 10A), and such fringeartifacts at the hologram plane will spread out to the wholereconstructed image and generate severe rainbow artifacts, as can beseen in FIGS. 10B, 10E. D-PSR results for the same samples (FIGS. 10C,10F) show significant improvements and suppression of such colorartifacts, in addition to having much better spatial resolution comparedto interpolation based de-multiplexing results shown in FIGS. 10B, 10E.

Next, color-stained Papanicolaou smears (Pap smears) were imaged thatare frequently used for screening of cervical cancer in order to comparethe color imaging performance of D-PSR against some of the previouslyreported holographic color imaging techniques, including sequential RGBimaging and YUV color-space averaging. As illustrated in theexperimental comparison that is provided in FIGS. 11A-11M, D-PSR has avery similar color imaging performance compared to sequential RGBimaging; however, by benefiting from simultaneous multi-wavelengthillumination, D-PSR uses 3-fold less number of measurements compared tosequential color imaging, which makes it much more data efficient andfaster. YUV color-space averaging, on the other hand, acquires a similarnumber of raw measurements/holograms compared to D-PSR, i.e., N+3 vs. N,respectively, where N is the number of raw measurements that D-PSR uses.However, the color imaging performance of YUV color-space averagingtechnique is inferior to D-PSR as it shows color bias and artifacts,also causing color leakage at the borders of rapid spatial transitionsas illustrated in FIG. 11C, 11G, 11K. In the last column of images,namely, FIGS. 11E, 11I, and 11M, microscopic images of the same sampleregions of interest taken using a 40×0.75NA objective-lens are alsoshown for comparison. Note that the lens-based microscope images areblurred in some regions because of the limited depth-of-focus comparedto lens-free microscopy images. Furthermore, to emphasize the large FOVadvantage of lens-free on-chip microscopy, typical FOVs of 40× and 20×objective-lenses are also shown in FIG. 11A.

It should also be noted that, in addition to 3-fold imaging speedimprovement and reduced number of measurements compared to sequentialcolor illumination, there are other reasons that sometimes simultaneousmulti-wavelength illumination is preferred and D-PSR could be applied.For example, in imaging flow-cytometry systems, specimens (e.g.,parasites or cells of interest) are constantly moving in a flow, and amotion-based PSR approach can be combined with D-PSR to get color imagesof the flowing micro-objects without the need for sequential multi-colorillumination, which would directly improve the flow rate and thethroughput of the imaging cytometer.

Finally, it is important to emphasize that the use a color (e.g., aBayer RGB) image sensor chip, as compared to a monochrome image sensor,has several advantages for holographic microscopy applications. First,color image sensors are much more cost-effective compared to theirmonochrome versions due to economies of scale and their massive adoptionin consumer electronics market, especially in mobile-phones. Second,most of these small pixel pitch CMOS image sensor chips, including theone that is used herein with ˜1.1 μm pixel size, are not available forsale in monochrome format, which limits the spatial resolution that onecan achieve using on-chip microscopy techniques with a monochrome chip.

Optimization of the Choice of Illumination Wavelengths in D-PSR

The optimal multi-wavelength illumination choice was analyzed based onthe spectral characteristics of the Bayer CMOS image sensor chip (SonyIMX85) that was used using a brute force search. It was assumed that themain sources of de-multiplexing error on a single pixel come from: (1)thermal noise of the sensor, and (2) quantization noise. It should benoted that if the three multiplexed wavelengths are chosen to be tooclose to each other, the cross-talk among channels will be significantand the de-multiplexing matrix will be almost singular, causing anysource of error (due to thermal noise and quantization noise) to besignificantly amplified.

As detailed in the flow-chart shown in FIG. 12, a brute force search wasperformed, where all the possible wavelength combinations were scannedfrom 400 nm to 700 nm at 1 nm step size and a de-multiplexing error wascalculated for each combination. For this calculation, it was firstassumed that the input signal for each channel follows a Gaussiandistribution with a mean of 1 and a standard deviation of 0.2. Then,using the measured/calibrated sensor response of the Bayer CFA, across-talk matrix was generated and the Bayer channel intensities werecalculated. Each channel intensity also included a thermal noise term(˜3.5%) and was quantized with a bit depth of 10 bits. Next, usingEquation (2), the de-multiplexing step was performed for all the RGBintensity points in the Gaussian distribution, the results of which arecompared to the input distribution to calculate an average error foreach channel. The total error for a given combination of illuminationwavelengths is taken as the maximum of these three mean errors arisingfrom R, G and B channels. Based on this combinatorial search, theoptimal illumination wavelength trio that has the smallestde-multiplexing error is found to be (456 nm, 570 nm, 651 nm). However,the gradient near this optimal point is actually quite small andtherefore a similar level of de-multiplexing error can be practicallyachieved over a large spectral range (see FIGS. 9A and 9B). For example,a region with a maximum de-multiplexing error of 6% (1.7× of thermalnoise level) is shown in FIG. 9A and it spans more than ˜50 nm for allthree Bayer channels. In FIG. 9B, the cross-sectional 1D-plots of thede-multiplexing error is shown near the choice ofmultiplexed-illumination wavelengths that was used in the experiments(i.e., 471, 532, 633 nm), confirming that the change of thede-multiplexing error is rather small over a relatively large spectralrange.

YUV Color-Space Averaging Method

One of the comparisons to the D-PSR technique is made using the YUVcolor-space averaging method. In this technique, the color informationis retrieved from three low resolution holograms at R, G and B colorchannels, which are then back-propagated to the sample plane, combinedand transformed into YUV color-space, and low-pass filtered by anaveraging window size of e.g., 10 pixels on the U and V channels to getrid of twin-image related rainbow artifacts of holographic imaging. Thehigh resolution (i.e., pixel super-resolved) Y channel, which requiresthe acquisition of N raw holograms (same as D-PSR), and the lowresolution U and V channels, which require the acquisition of three rawholograms, are then fused in the YUV color-space, and finally convertedinto RGB space to get a color image of the specimen.

Demosaicing induced holographic color artifacts that arise due tolimited spatial sampling at a Bayer CFA are significantly alleviated inD-PSR through the digital synthesis of spatially overlapping and muchsmaller effective pixels in each color channel. Furthermore, in D-PSRthe pixel-level spectral cross-talk of a Bayer CFA is compensated bydigital de-multiplexing. Compared to holographic color imaging usingsequential multi-wavelength illumination, this new approach takes 3-foldless number of raw holograms/measurements while also achieving a verysimilar color imaging performance. D-PSR can be broadly used forhigh-resolution holographic color imaging and microscopy applications,where wavelength-multiplexing is desired.

While embodiments of the present invention have been shown anddescribed, various modifications may be made without departing from thescope of the present invention. For example, while the method haslargely been described using a lens-free embodiment to obtain pixelsuper-resolution images, the method may also be implemented using alens, lens set, or lens module located within the optical path. Theinvention, therefore, should not be limited, except to the followingclaims, and their equivalents.

What is claimed is:
 1. A method of generating a color image of a samplecomprising: obtaining a plurality of low resolution holographic imagesof the sample using a color image sensor, the sample illuminatedsimultaneously by light containing multiple wavelengths or colors,wherein the illuminated sample casts sample holograms on the color imagesensor and wherein the plurality of low resolution holographic imagesare obtained by relative shifts in the x, y, and z direction between thesample holograms at a given axial plane and the color image sensor,wherein at least one lens, lens set, or lens module is located along anoptical path between the sample and color image sensor; generating apixel super-resolved hologram of the sample using the plurality oflow-resolution holographic images obtained by simultaneous illuminationof the sample by light from the multiple wavelengths or colors;generating de-multiplexed pixel super-resolved holograms at individualwavelengths or colors using the pixel super-resolved hologram obtainedfrom the simultaneous illumination of the sample by; retrieving phaseinformation from the de-multiplexed holograms at each of the wavelengthsor colors using a phase retrieval algorithm to obtain a complex hologramat the wavelengths or colors; and digitally back-propagating andreconstructing the complex hologram for each of the wavelengths orcolors to a sample plane to generate the color image of the sample bycombining the reconstruction results of each of the wavelengths orcolors.
 2. The method of claim 1, wherein the color image sensorcomprises a periodic array of pixels, where pixels of the array havemultiple wavelength channels or transmission spectra.
 3. The method ofclaim 2, wherein the wavelength channels or transmission spectra aregenerated by a color filter.
 4. The method of claim 3, wherein the CFAis selected from the group consisting of a Bayer filter, RGBE filter,CYYM filter, CYGM filter, RGBW Bayer filter, and RGBW filter.
 5. Themethod of claim 1, further comprising performing saturation correctionon the acquired holograms corresponding to the wavelengths or colors. 6.The method of claim 1, further comprising white-balancing thede-multiplexed holograms corresponding to the wavelengths or colors. 7.The method of claim 1, wherein the sample is illuminated simultaneouslyby light from multiple illumination sources.
 8. The method of claim 1,wherein the sample is illuminated simultaneously from a broadband lightsource that is filtered into a plurality of bands.
 9. The method ofclaim 8, wherein the plurality of bands have different bandwidths. 10.The method of claim 1, wherein the sample comprises a biological sample.11. The method of claim 1, wherein the phase retrieval algorithmcomprises a multi-height (z) phase retrieval algorithm that uses one ofthe pixel super-resolved holograms at one of the wavelengths or colorsand iteratively propagates it to a next (z) measurement plane where theamplitude of the field is partially or entirely replaced with themeasured amplitude and continues until a last measurement plane (z) isreached.
 12. A system for generating color images of a samplecomprising: an optically transparent sample holder configured to holdthe sample thereon; at least one light source configured tosimultaneously output multiple wavelengths or colors on a first side ofthe sample holder; a color image sensor disposed on a second side of thesample holder; one or more processors configured to execute imageprocessing software thereon, the image processing software: obtaining aplurality of low-resolution holographic images of the sample using thecolor image sensor, wherein the simultaneously illuminated sample castssample holograms on the color image sensor and wherein the plurality oflow-resolution holographic images are obtained by relative x, y, and zdirectional shifts between sample holograms and the color image sensor;generating a pixel super-resolved hologram of the sample using theplurality of low-resolution holographic images obtained by simultaneousillumination of the sample by light from the multiple wavelengths orcolors; generating de-multiplexed pixel super-resolved holograms at eachof the at least multiple wavelengths or colors using the pixelsuper-resolved hologram obtained from the simultaneous illumination ofthe sample; retrieving phase information from the de-multiplexedholograms at each of the at multiple wavelengths or colors using a phaseretrieval algorithm to obtain a complex hologram at each of the multiplewavelengths or colors; and digitally back-propagating and reconstructingthe complex hologram for each of the multiple wavelengths or colors to asample plane to generate the color image of the sample by combining thereconstruction results of each of the multiple wavelengths or colors.13. The system of claim 12, wherein the color image sensor comprises aperiodic array of pixels, where pixels of the array have multiplewavelength channels or transmission spectra.
 14. The system of claim 12,wherein at least one lens, lens set, or lens module is located along anoptical path between the sample and color image sensor.
 15. The systemof claim 12, wherein the sample is illuminated simultaneously by lightfrom multiple illumination sources.
 16. The system of claim 12, whereinthe sample is illuminated simultaneously from a broadband light sourcethat is filtered into a plurality of bands.
 17. The system of claim 16,wherein the plurality of bands have different bandwidths.
 18. The systemof claim 12, wherein the relative x, y, and z directional shifts areperformed by translation stage coupled to the color image sensor. 19.The system of claim 12, wherein the relative x, y, and z directionalshifts are performed by translation stage coupled to the sample holder.20. The system of claim 12, wherein the relative x, y directional shiftsare performed by actuating different light sources.